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ERIC 



As the technology and tools to use the Internet become more streamlined, the option of using the 
World Wide Web to conduct survey research becomes more and more attractive. This method 
requires fewer resources and provides faster responses than the traditional paper and pencil 
methods. However, new methodologies create questions about response consistency, response 
rates, and technical issues related to the technology. The Internet’s potential for academic and 
applied research has recently begun to be acknowledged and assessed, but comparisons with 
more traditional survey methods must be conducted (Sheehan & Hoy, 1999; Sussman, 2001; 

Yun & Trumbo, 2000). 

As a consultant, this researcher has the opportunity to conduct a number of surveys for 
professional organizations and academicians each year. In 2001, a survey for a nonprofit library 
network in the Southeastern United States was conducted, using a database of over 1400 names. 
Half of the names in the database had mailing addresses and e-mail addresses, while the other 
half had only mailing addresses. In order to expedite the process and save on costs, two methods 
were used to contact the sample. One was a traditional three-wave mailing procedure. The other 
used e-mail with a link to a website where the same questionnaire was located. 

The survey was not conducted as an experiment. The data received from the respondents 
were used to create reports for the client. After completion of the project, the idea of using the 
data for this analysis was conceived. No individual demographic data was obtained on the 
respondents. Only the institutions’ characteristics were of importance to the client. Although the 
results discussed in the remainder of this article were not obtained under experimental 
conditions, the opportunity to compare the data from the two survey methods proved irresistible. 

Definition of Terms 

With the surge in computerization in the last decade, a clarification of terms is necessary 
when discussing different survey methods. 

A traditional postal survey involves a questionnaire, a cover letter, and a return envelope. 
To maximize response rate, many postal surveys include a follow-up postcard, and at least one 
additional full mailing of the survey. Some postal surveys include pre-notification letters and 
more follow-ups. Data entry is necessary to convert the responses on paper to an electronic 
database. 

E-mail surveys are simpler to compose than Web-based surveys, but they are more 
limited to their visual stimulation and interaction capabilities. At the present, an e-mail survey is 
no more than a text message that requires no skills above those used to compose and send a 
message to a friend. They cannot provide for skip patterns and other intensive programming 
options (Dillman, 2000). Again, data entry is necessary to convert the responses in the e-mail to 
an electronic database. 

Web-based surveys share the same type of computer to computer communication over the 
Internet with e-mail surveys, but they “provide survey capabilities far beyond those available for 
any other type of self-administered questionnaire” (Dillman, 2000, p. 354). Extensive and 
difficult skip patterns can be designed so that they are invisible to the respondent. Color, 
animation, and audio can be incorporated. Coding and data management is all handled 
electronically and provides a complete database ready for analysis at survey end, thus removing 
one source of error — human data entry error. 
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Relevant Literature 



Response Rates 

Response rates to e-mail surveys have significantly decreased since 1986 (Sheehan, 

2001). In 1986, results from the first e-mail survey were published in Public Opinion Quarterly 
(Kiesler & Sproull, 1986). Moreover, in the past decade, the popular press has reported that 
response rates are declining for all types and manner of surveys (Bickard «fe Schmittlein, 1999). 

Matz (1999) reported response rates to a Web survey were not as high as traditional paper 
surveys. Sheehan (2001) attempted to identify as many e-mail surveys done for academic 
purposes as possible and only 3 1 surveys could be identified that contained sufficient data to 
conduct an analysis of response rate. The results suggested that e-mail survey response rates 
have followed the pattern of survey response rates overall in the United States. Sheehan posed 
several questions about the value of e-mail surveys: (a) Would a link to a Web site be easier than 
other e-mail methods? (b) Do we need to provide multiple options to increase response rates? 
and (c) Should we continue to even try to evaluate e-mail surveys as a viable data collection 
method or should we focus our attention on increasing reliability of surveys on the Web? 
Although Web-based surveying raises a host of new methodological issues, Dillman (2000) 
argues that “no other method of collecting survey data . . . offers so much potential for so little 
cost” (p. 400). 

Response Consistency 

The quality of data among survey methods is an issue for researchers. Electronic surveys 
are increasingly more common and research comparing electronic vs. postal surveys is beginning 
to confirm that electronic survey content results may be no different than postal survey content 
results (Andrews, Nonnecke, & Preece, 2001). Some studies provide evidence that the e-mail 
survey has more non-response items than paper-and-pencil surveys (Bachmann &. Elfrink, 1996; 
Sproull, 1986). Others argue that there is minimal difference between approaches (King &, Miles, 
1995; Matz, 1999; Tse, 1998) and that e-mail methods generate fewer non-response items 
(Schaefer & Dillman, 1998). Stanton (1998) found that the number of missing data points 
decreased in the Internet survey, but found no difference in variability or internal patterning of 
the data. These findings suggest that “e-research” is comparable to the traditional mail-in method 
and improves data collection. 

Yun and Trumbo (2000) reported that respondent characteristics differed (between 
traditional paper-and-pencil and Web-based surveys) in the variables that are explicitly and 
implicitly related to the use of communication technology. Buchanan and Smith (1999) 
compared the responses on a paper and pen self-monitoring scale to a Web-based equivalent. 
They found comparable results between the groups and that the Web-based participants had 
slightly higher reliability than the comparison group. 

Yun and Trumbo (2000) did not observe significant influences of survey mode (postal, e- 
mail, and Web). The results of their study led them to report that the differences detected in the 
response groups indicate that using multi-mode survey techniques improved the 
representativeness of the samples without biasing other results. 
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Issue salience 



Salience of an issue to the sampled population has been found to have a strong positive 
correlation with response rate for postal, e-mail, and Web-based surveys (Sheehan & McMillan, 
1999; Watt, 1999). Salience has been defined as the association of importance and/or timeliness 
with a specific topic (Martin, 1994). Heberlein and Baumgartner (1978) reported that issue 
salience had more influence on mail survey response rates than other factors such as respondent 
contact and monetary incentives. Bean and Roszkowski (1995) suggested salience has more 
influence on response rate than survey length. 

Technical problems 

Another potential problem for electronic surveys is that of multiple submissions. If the 
researcher designs the survey with an anonymous response function it is almost impossible to 
detect multiple submissions. The problem is even more complex with the Web survey. 
Researchers may assume that Web site visits from the same IP address are from a single person, 
but in reality it may not be true (Smith, 1997). Using different methods of identifying the 
respondent and guaranteeing confidentiality is key to combating some of the more esoteric 
aspects of conducting a Web-based survey. 

Current Study 



Procedure 

After completing a survey in 2001 for a nonprofit library network in the Southeastern 
United States this researcher used the database to study response rates and response consistency 
between two survey methods. Information on more than 1400 potential respondents had been 
collected from the network’s database and four other library databases. Half of the librarians’ 
had contact information that included an e-mail address. A traditional mailing procedure was 
used to obtain responses to a 4-page printed questionnaire from those individuals in institutions 
who had provided only postal contact information (Appendix A). 

For those individuals with an e-mail address, the survey was conducted by e-mail, with a 
link to a Web-based questionnaire created from the printed questionnaire. The Web-based 
questionnaire was identical to the print survey. The hyperlink in the e-mail cover letters 
(Appendix B) led the respondent to the questionnaire at the researcher’s website, which is hosted 
by Earthlink. The Web-based questionnaire was developed using Microsoft software, FrontPage, 
with a link to an Access database. Radio buttons and drop-down boxes were used for answer 
options. Each question was programmed so that the answer options were adhered to (i.e., only 
one choice could be selected if only one answer was required). Each respondent was required to 
provide an e-mail address before they could submit their responses. 

The survey was conducted in October and November 2001. The Web-based procedure (« 
= 699) included two e-mail contacts, three weeks apart, with potential respondents. The mail 
procedure (« = 730) included a first full mailing, a postcard reminder one week later, and a 
second full mailing to nonrespondents two weeks after the postcard. Responses to the Web-based 
questionnaire were electronically downloaded into a database, while the responses from the 
questionnaires returned by mail were hand entered into a database. 
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Respondents to the electronic questionnaire were asked to provide their e-mail address so 
no further follow-ups would be made to them. Their e-mail addresses were hand filtered from the 
database as their response was received. Mailed questionnaires were coded to track the response 
rate and reduce the number of pieces of follow-up mail. Their mailing addresses were hand 
filtered from the database as their response was received. Respondents were assured 
confidentiality by the research firm and the researcher conducting the survey. The client (the 
library network) received a report with only aggregate data. All analyses of the response sets 
were conducted using SPSS. 

Response Rates 

The researcher subjectively rated the salience value for this survey at 2 or 3 on a scale of 
l(low) to 5 (high). The topic (preservation) is an important but often under-funded (therefore, 
neglected) aspect of library management. Additionally, all the members of the sample were 
librarians, but not all were employed at institutions that were members of the library network 
(the sponsoring agency). 

Response rate, which has been linked to issue salience, was not as high for this survey as 
for previous surveys conducted for the library network (Table 1). Refusals (those individuals 
who elected not to participate after contact had been made) and bad postal addresses reduced the 
postal database from 730 to 709. Responses numbered 192 for a response rate of 27% for the 
postal survey. Refusals and bad e-mail addresses reduced the Web-based database from 699 to 
616. Responses numbered 186 for a response rate of 30% for the Web-based survey. No 
information was collected as to response rate after each wave of the postal or electronic 
reminders. 

Statistically, there was no difference in the response rate of the two methods. This finding 
does not support that of Shannon and Bradshaw (2002) who found that electronic surveys did not 
generate a response rate as high as mail surveys. The larger number of bad addresses from the 
Web-based procedure is consistent with a number of studies using electronic surveys (Couper, 
Blair, & Triplett, 1997; Sheehen & Hoy, 1999; Smith, 1997). 

Table 1. Response Rates 





Original 

database 


Bad addresses/ 
refiisals 


Responses 


Response 

rate 




Web-based 


699 


83 


186 


.30 




Postal 


730 


21 


192 


.27 


2.94 (ns) 



Response Consistency 

The librarians who participated in the survey responded to a number of items concerning 
preservation issues at their institutions. Of particular importance to the client (the library 
network) was the representation of the respondents across the region represented by the library 
network. Additional institutional demographic variables were staff size (FTE) of the institution 
and the cost of preservation activities in the previous fiscal year. An analysis of the responses by 
survey method showed no differences between the two methods in the representation from the 1 1 
states in the region (Table 2). No differences were found in the previous fiscal year’s cost of 
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preservation activities. The size of the staff (FTE) was significantly different between the two 
methods. The individuals responding to the Web-based questionnaire reported that their 
institutions were manned by larger staffs than those responding to the postal questionnaire. 



Table 2. Representation of Responses between Methods 





Web-based 


Postal 








n 


% 


n 


% 




P 


State 














Alabama 


19 


3.9 


7 


3.9 






Arkansas 


3 


1.6 


3 


1.7 






Florida 


23 


12.0 


24 


13.4 






Georgia 


22 


11.5 


24 


13.4 






Kentucky 


15 


7.8 


18 


10.1 






Louisiana 


14 


1.3 


8 


4.5 






Mississippi 


9 


4.7 


12 


6.7 






North Carolina 


23 


15.1 


24 


13.4 






South Carolina 


15 


7.8 


19 


10.6 






Tennessee 


18 


3.4 


9 


5.0 






Virginia 


25 


13.0 


31 


17.3 


12.13 


.28 




M 


SD 


M 


SD 


t 


P 


Cost preservation activities in last year 


30581 


86067 


37017 


138462 


-.47 


.64 


Number of FTE staff 


49.46 


113.10 


17.34 


49.12 


3.43 


<.01 



Differences occurred in the internal consistency of the several sets of Likert scaled items 
(Table 3). Across the board, the reliability of the scales from the postal respondents was higher 
than those from the Web-based respondents. This finding is not consistent with results from 
Buchanan and Smith (1999) who found Web-based participants had slightly higher reliability. 

Table 3. Reliabilities of Selected Sections of Questionnaire 



Section of Questionnaire number Cronbach’s alpha 

of items Web-based Postal 



Question 1 (now) 


9 


.73 


.80 


Question 1 (3 years in future) 


9 


.73 


.79 


Question 2 


9 


.83 


.88 


Question 3 


9 


.83 


.90 



Items that related to technology involving the Internet were also analyzed for the 
consistency of the responses between the two methods. Those who responded electronically were 
more likely to chose Web-based training, but were as interested as their colleagues who 
responded by mail in Internet publications, Internet training, or information gained from the 
Internet for their preservation needs (Table 4). This finding is not consistent with the general 
findings of Shannon and Bradshaw (2002) who reported that respondents to an electronic survey 
had more favorable attitudes toward technology than respondents to a postal survey. 
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Table 4. Comparison of Responses Related to Technology 





Web-based 


Postal 








n 


% 


n 


% 


P 


Web-based training (% prefer) 
Use Internet publications about 


89 


45.2 


65 


32.9 


3.36 


.07 


preservation needs (% who do) 
Use Internet to gain access to training 


99 


50.3 


84 


46.2 


.64 


.43 


for preservation needs (% who do) 
Use Internet to gain information about 


34 


17.3 


41 


22.5 


1.65 


.20 


preservation needs (% who do) 


142 


72.1 


120 


65.9 


1.38 


.20 



The questionnaire contained 1 1 0 items. Missing items were summed for each respondent 
(Table 5). The amount of missing data was significantly greater from the postal questionnaire 
than from the Web-based questionnaire. This finding is consistent with those of Schaefer and 
Dillman (1998) who reported that e-mail methods generate fewer non-response items. 



Table 5. Missing Data 





Range 


M 


SD 


t p 


Web-based 


0-51 


4.50 


7.00 




Postal 


0-61 


6.83 


10.71 


-2.54 .01 



Conclusions 

The opportunity to analyze data from two different survey methods provided important 
information about response rates and response consistency. The response rates of the two 
methods were similar. Attitudes toward questions concerning use of the Internet did not show a 
difference between the two groups of respondents. The likelihood of respondents of the mail 
survey using the Internet for training and accessing information about library preservation issues 
was the same as respondents who responded to the Web-based survey. 

Librarians who responded to the postal survey were more likely to be from a small 
institution (with smaller FTEs). This could be an indication of why there were no e-mail 
addresses for these individuals. Smaller libraries tend to have less technology available for their 
staff, thus reducing the possibility of each member having a personal electronic address. 

Although the reliability estimates from each method were well within acceptable ranges, 
the postal survey obtained higher values than the Web-based method. Results from researchers 
are mixed on this subject (Buchanan & Smith, 1999; Stanton, 1998). Further research into this 
phenomenon is warranted. 

The results of the analysis of the different methods show that the amount of missing data 
is significantly reduced when the Web-based survey method is used. Although the Web-based 
survey did not specifically require respondents to complete all items, there were built-in 
safeguards to protect against providing more than one answer per item. Additionally, the ease in 
using the drop-down boxes and radio buttons may have provided incentive to complete the 



questionnaire in its entirety. This finding is consistent with that reported by Schaefer and 
Dillman (1998). As research continues into the way people respond to paper-and-pencil and 
electronic questionnaires, this phenomenon may be explained. 

The use of e-mail to link potential respondents to a Web-based questionnaire produced a 
response rate similar to a traditional postal survey. Although slight differences occurred in 
internal reliability, the values were not depressed enough to discourage the use of Web-based 
surveys. Additionally, it can be argued that less missing data in the responses from the Web- 
based survey compensates for the lower reliability. Organizations and institutions with well- 
developed databases of potential respondents should feel confident that responses from an 
electronic survey using e-mail contact with a hyperlink to a Web-based questionnaire will 
produce faster, less expensive, and comparable data than a traditional postal survey. 
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SOLINET Preservation 2001 



1. How important are these preservation activities at your institution? 





Very 

important 


Now? 


Not 

important 


In three years? 

Very Not 

important important 


Disaster planning (or preparedness) 


1 


2 


3 


4 


1 


2 


3 


4 


Environmental monitoring 


1 


2 


3 


4 


1 


2 


3 


4 


Pest and mold management 


1 


2 


3 


4 


1 


2 


3 


4 


Collections conservation (book repair) 


1 


2 


3 


4 


1 


2 


3 


4 


Conservation of rare and special collections 


1 


2 


3 


4 


1 


2 


3 


4 


Library binding 


1 


2 


3 


4 


1 


2 


3 


4 


Microfilming 


1 


2 


3 


4 


1 


2 


3 


4 


Photocopying for replacement 


1 


2 


3 


4 


1 


2 


3 


4 


Digital reformatting 


1 


2 


3 


4 


1 


2 


3 


4 



2. Does your institution need to improve activity in these areas? 




Yes, it needs 


No, it is 


improvement 


adequate 



Disaster planning (or preparedness) 


1 2 


3 


4 


Environmental monitoring 


1 2 


3 


4 


Pest and mold management 


1 2 


3 


4 


Collections conservation (book repair) 


1 2 


3 


4 


Conservation of rare and special collections 


1 2 


3 


4 


Library binding 


1 2 


3 


4 


Microfilming 


1 2 


3 


4 


Photocopying for replacement 


1 2 


3 


4 


Digital reformatting 


1 2 


3 


4 



3. What is your staff training/education need in these preservation activities? 



Very much Not 

a need a need 



Disaster planning (or preparedness) 


1 


2 


3 


4 


Environmental monitoring 


1 


2 


3 


4 


Pest and mold management 


1 


2 


3 


4 


Collections conservation (book repair) 


1 


2 


3 


4 


Conservation of rare and special collections 


1 


2 


3 


4 


Library binding 


1 


2 


3 


4 


Microfilming 


1 


2 


3 


4 


Photocopying for replacement 


1 


2 


3 


4 


Digital reformatting 


1 


2 


3 


4 
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SOLINET Preservation 2001 



4. Which of the following does your institution currently need? {Check all that apply) 

a written disaster plan 

building renovation or construction 

mold control or prevention 

improved environmental control 

patron education on preservation 

reformatting of brittle materials (books, manuscripts, etc.) 

re-housing of archival collections 

preservation of electronic files 

preservation assessment of tlie institution and/or individual collections 

none 



5. What is keeping you from completing or obtaining any of the preservation activities you 
checked? 



6. What is your preferred method for training? {Check all that apply) 

one-day workshop 

two-day workshop 

conference program 

web-based training 

7. Within the last year how often did you or your staff participate in the following training 
activities? 



one-day workshops 


Never 


once 


twice 


3 times 


> 3 times 


two-day workshops 


Never 


once 


twice 


3 times 


> 3 times 


conference programs 


Never 


once 


twice 


3 times 


> 3 times 


web-based training 


Never 


once 


twice 


3 times 


> 3 times 



8. Does your institution pay for you to attend training events? 

No 

Yes What does the institution pay? (CA^cA a// /Aa/ app/j;) 

all of the registration fee 

part of the registration fee 

all travel costs 

part of the travel costs 

9. What are the barriers to you and your staff obtaining training? {Check all that apply) 

cannot travel 

limited staff 

no travel funds 

no training funds 

other {specify) 

none 



10. What is your institution’s annual budget for staff training and development? $ 
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SOLINET Preservation 2001 



11. In what format do you get information about preservation now? {Check all that apply) 

print publications 

electronic publications 

video publications 

attend a training event 

attend a program presentation at a conference 

take a web-based training program 

Do not get information about preservation 

12. Do you know about SOLINET’s Preservation Field Services? 

No 

Yes How did you find out about them? {Check all that apply) 

referral 

website 

training event 

conference booth 

pub lication (s) 

SOLINET brochure 

other (specify) 

13. Have you used SOLINET’s Preservation Field Services in the past? 

1) No 

2) Yes What is the most recent year you used the services? 

What services did you use? {Check all that apply) 

training 

information and referral 

publications 

consulting 

audio-visual loan service 



14. Have you ever hired a preservation consultant from somewhere other than SOLINET? 

1) No 

2) Yes What is the most recent year? 

What did you pay? $ 

How many days did the consultant stay on-site? 

Did you receive a written report? 

1) No 

2) Yes 

15. Has your institution purchased publications on preservation topics in the last two years? 

1) No 

2) Yes 

16. Do you use the Internet to access any of the following preservation needs? {Check all that apply) 

publications 

training 

information 

Do not use tlie Internet for preservation needs 

17. Do you need (or would you like) preservation materials and activities in Spanish? 

1) No 

2) Yes 
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18. Approximately how much money did your institution spend on preservation activities in the last 
fiscal year? 



19. How will your institution’s current fiscal year’s budget for preservation change from the above 
figure? {Select only one) 

1) Increase by more than 50% 

2) Increase by less than 50% 

3) Decrease by more than 50% 

4) Decrease by less than 50% 

5) No change 

20. What is the number of full-time equivalent (FTE) staff in your institution? 

FTE 

21. What does your institution need to develop and sustain your preservation activities? 



Thank you for your participation! 

Please return completed questionnaire to 
Idleman & Associates 
P.O. Box 729 
Decatur, Georgia 30031 
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Appendix B 
Web-based Survey 



Text of first e-mail message 
Subject: Preservation Field Services Needs Your Input 
Dear Colleague: 

Established in 1985, SOLINET's Preservation Field Services works to improve the 
ability of institutions to preserve and provide access to their collections. With funding 
from the National Endowment for the Humanities, Preservation Field Services offers 
assistance to documentary repositories throughout the Southeast, including libraries, 
archives, historical societies, and museums. 

The Andrew W. Mellon Foundation recently provided a grant to Preservation Field 
Services to develop a business plan. This survey will help provide information for that 
plan. The questionnaire seeks information from current and potential users of 
Preservation Field Services, to help identify needs and possible future services. 

You are an important source of the information we are seeking from your institution. 
Would you spend 10-15 minutes completing the on-line questionnaire? The 
independent research firm of Idleman & Associates is collecting the data to insure the 
confidentiality of your responses. One feature of the questionnaire is a request for your 
e-mail address. To furnish Idleman & Associates with this information affords a more 
efficient survey process and you will not receive further reminders about the 
questionnaire. Again, your responses are confidential and the request for your e-mail 
address is only to aid tracking of responses. 

Please feel free to contact me at 404.892.0943 or Idleman & Associates at 404.815.7280 
with any questions you may have about this study. Thank you for helping us provide the 
services that your institution needs to perform its preservation activities. 

Julie Arnott, Director, Preservation Field Services 

Click here for the PRESERVATION SURVEY 
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Text of second e-mail contact 



Dear Colleague: 

Last month we asked you to provide information about your preservation needs. We are 
seeking information from current and potential users of Preservation Field Services to 
help identify needs and possible future services. As of today, we have not received 
your completed questionnaire. You are an important source of the information we are 
seeking from your institution, so we are again asking for your help. Would you spend 
10 minutes completing the on-line questionnaire? 

Established in 1985, SOLESIET's Preservation Field Services works to improve the 
ability of institutions to preserve and provide access to their collections. With funding 
from the National Endowment for the Humanities, Preservation Field Services offers 
assistance to documentary repositories throughout the Southeast, including libraries, 
archives, historical societies, and museums. The Andrew W. Mellon Foundation 
recently provided a grant to Preservation Field Services to develop a business plan. This 
survey will help provide information for that plan. 

The independent research firm of Idleman & Associates is collecting the data to insure 
the confidentiality of your responses. One feature of the questionnaire is a request for 
your e-mail address. To furnish Idleman & Associates with this information affords a 
more efficient survey process and you will not receive further reminders about the 
questionnaire. Again, your responses are confidential and the request for your e-mail 
address is only to aid tracking of responses. 

Please feel free to contact me at 404.892.0943 or Idleman & Associates at 404.815.7280 
with any questions you may have about this study. Thank you for helping us provide the 
services that your institution needs to perform its preservation activities. 

Julie Arnott, Director, Preservation Field Services 

Please go to www.idleman.com/solinet for the survey 
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